V0.9.1/evals - #39

Add portable evaluation runner framework with support for multiple harnesses: Codex, OpenCode, and deterministic Fake runner. Includes contract schemas, runner adapters, result bridging, and conformance tests. Refactor eval preparation and report generation scripts to support the new runner abstraction.

Add isolation capability assessment and preflight validation to detect unsupported runner configurations. Implement Cline runner adapter for eval orchestration. Enhance common runner utilities with sandbox path mapping, external command versioning, and environment management. Expand conformance tests with additional event fixtures.

Add GitHub Copilot CLI as a supported Eval Runner alongside Cline, Codex, and OpenCode. The runner handles Copilot-specific authentication (GitHub tokens), isolation constraints, and JSONL-based event output parsing. Includes conformance tests and reference documentation for the runner protocol compliance.

Extend prepare-skill-evals.ps1 and validate-skill-templates.ps1 to support GitHub Copilot runner configuration and execution profile selection. Enables dynamic runner resolution and integration with the new GitHub Copilot CLI evaluation path.

Establish GitHub Copilot with claude-haiku-4.5 as the Codebelt reference evaluation configuration for economical and stable comparison across evals. Update eval isolation terminology from 'hermetic' to 'isolated' and clarify Windows support as a first-class pragmatic evaluation target. Document filesystem confinement as a confidence enhancement rather than a universal prerequisite, enabling broader platform coverage.

Modernize the GitHub Copilot eval runner to deliver the prepared prompt through stdin instead of the --prompt argument. This improves byte fidelity and supports larger prompts without hitting command-line length limits.

Update authentication handling to follow Copilot's normal precedence: explicit COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN environment variables, then OS keychain, then GitHub CLI fallback through GH_CONFIG_DIR. Remove --no-custom-instructions so repository-owned instructions remain visible to both paired arms. Add COPILOT_CACHE_HOME isolation. Refine --secret-env-vars to filter all listed token variables from shell and MCP child environments. Update preflight to report conditional native keychain readiness without requiring a live model request.

Extend runner conformance tests to validate stdin prompt delivery with byte-exact verification, multiple token variables, repository instruction visibility, and authentication source detection.

Change from defensive directory creation to explicit validation. The conformance test now requires the output parent directory to already exist and exits with a clear error message if it doesn't, improving error clarity when the directory structure is misconfigured.

Restructured AGENTS.md Skill Authoring section for clarity, brevity, and progressive disclosure. Updated README.md skill catalog to note that descriptions are lean activation metadata. Optimized all 21 repo-managed skill descriptions to be concise, trigger-oriented metadata following the new authoring standard. Changes align description quality with specification requirements and improve skill discoverability.

Added Layered Capability Classification section to distinguish classification of independently selectable adapters and runners from their parent directory or framework. This ensures that new child adapters are correctly classified as Added rather than grouped with pre-existing framework changes. Added comprehensive eval case covering framework refinement, new adapter introduction, defect repair, and layered state classification to validate the enhanced guidance.

Extended git-keep-a-changelog skill validation to verify the presence and correctness of the new Layered Capability Classification section and its guidance on adapter/runner boundary classification. Added checks for section presence, guidance on child adapter states, refinement classification, and avoidance of repeat classification patterns.

Regenerated [0.9.1] release entry from current git state (2026-08-22) to capture all commits through HEAD, including GitHub Copilot CLI runner support, Cline runner addition, skill description optimization, and AGENTS.md authoring guidance restructuring. Updated release highlight, Added/Changed/Fixed sections, and compare link to reflect the complete v0.9.1 release scope.

Updated repository guidelines, contributor expectations, and evaluation process documentation to distinguish Eval Runners, Orchestrators, and Graders while clarifying the explicit external-handoff boundary. All 21 repo-managed skill descriptions refactored for trigger-oriented activation metadata following progressive disclosure. Improved section organization in AGENTS.md for skill authoring, form handling, and dynamic defaults.

Added entry documenting the harness model resolution workflow, removal of the redundant provider field from execution-profile.json, and introduction of scripts/Get-HarnessModels.ps1 for current model discovery with Codebelt Reference verification and platform-specific filtering.

Modernized eval runner implementations across all harness types (GitHub Copilot, Codex, OpenCode, Cline) with updated contract schemas. Added scripts/Get-HarnessModels.ps1 for discovering current model selectors per harness with Codebelt Reference verification and platform-specific filtering. Improved prepare-skill-evals.ps1 to resolve Harness + Model before package generation, updated validation and conformance testing, and refined execution-profile.json to remove redundant provider field while treating model selectors as runner-native opaque strings.

Added manifest-paths.ps1 and bridge-manifest-results.ps1 for safe manifest path resolution and result artifact bridging. Updated prepare-skill-evals.ps1 with improved error handling and result collection. Enhanced generate-eval-report.ps1 for report generation and grading workflow. Expanded validate-skill-templates.ps1 with comprehensive fixture validation. Improved conformance testing in test-runner-conformance.ps1 to cover all eval runner contracts.

Update repository guidance in AGENTS.md, CONTRIBUTING.md, and README.md to explicitly clarify that the eval completion gate must be satisfied before a package is presented as successfully completed. Incomplete or unrun evaluation packages must be flagged as incomplete, not misrepresented as passing.

Update the generated README.md content in prepare-skill-evals.ps1 to reflect the clarified eval completion gate behavior. Packages generated by this script now include text clarifying that incomplete or unrun arms must be reported as such and that -CollectResults exits non-zero when the completion gate is not satisfied.

Implements deterministic queue and state management for eval workers. Adds native delegation surface with full-capability and model-lock guarantees. Each eval arm executes in an isolated harness-native worker with working-directory isolation, result capture, and fresh session constraints. Supports concurrent worker coordination with configurable capacity slots and rejection handling without eval attempt increment.

Adds native worker terminal evidence requirements and validation logic. Enhances delegation capability assessment with status tracking (supported/conditional/unsupported). Implements terminal evidence collection in orchestration tests including session tracking, model observation, directory isolation verification, and prompt fidelity checks. Updates result bridging to validate native worker evidence when required. Expands runner descriptor validation and adds comprehensive terminal evidence audit trail.

Update SKILL.md and evals.json to explicitly clarify that when the user provides 'yolo' or 'auto' in an explicit commit request, the skill should complete the workflow in the same turn after required checks pass, rather than awaiting additional approval or returning a pending plan.

Add validation assertions to ensure the git-visual-commits SKILL.md contains required phrases about completing the commit workflow in the same turn and that the plan summary is status output rather than a review request.

Update Git Operations Safeguards section to explicitly state that yolo or auto on an explicit commit request counts as approval to complete the commit workflow in the same turn, and clarifies that such approval must be attached to the same explicit commit request rather than treated as a standalone approval modifier.

…ance

…into v0.9.1/evals

Preserve runner-reported terminal status exactly and record evidence validation separately. Fix requires different evidence requirements for completed vs non-success terminals to accept honest timed-out, failed, or cancelled results without forcing fabricated responses.

Verify that ledger preserves raw runner status and evidence_validation is recorded separately. Test mixed terminal outcomes (timed_out, failed, completed, cancelled) and OpenCode timeout handling.

Add infrastructure to aggregate Phase 1 orchestration outcomes and validate that all expected evaluation arms complete. Extract and refactor orchestration state inspection into reusable aggregation functions. Update fanout summary generation to track terminal, completed, failed, timed-out, cancelled, incompatible, and evidence-validation-failed counts. Include Phase 1 success test and fail-closed gate in orchestration path. Add deterministic regression test suite for Phase 1 aggregate calculations and manifest result validation for terminal non-completed states.

Adapt prepare-skill-evals.ps1 and validate-skill-templates.ps1 to integrate with the new durable Phase 1 supervisor model. Updates include support for the bounded controller surface, improved timeout handling, and alignment with runner-owned phase lifecycle expectations.

Update AGENTS.md to clarify the runner-owned Phase 1 orchestration workflow, replacing the previous fan-out helper language with explicit control-runner-owned-phase1.ps1 controller invocation. Emphasize that the external orchestrator repeats the same bounded idempotent call, the durable supervisor manages preflight and fan-out internally, and Phase 1 fails closed if the supervisor dies without a terminal result. Minor updates to CONTRIBUTING.md and README.md for consistency.

Extract phase1 control logic into a shared common module (phase1-control-common.ps1) and enhance process isolation with Windows Job Object validation. Add Windows interop layer for process/job management, ownership state tracking, and lifecycle verification. Refactor control-runner-owned-phase1.ps1 to leverage shared helpers. Enhance supervise-runner-owned-phase1.ps1 ownership state detection. Add Windows Job Object test harness to test-phase1-controller-lifecycle.ps1.

Add comprehensive test coverage for nested Windows Job Object breakaway detection. Includes new test cases validating that Phase 1 controllers properly handle scenarios where breakaway is requested but forbidden by an outer job, ensuring durable supervisor independence is strictly enforced. Updates test assertions and harness property references to support enhanced validation logic.

Model-free harness probes (--version, --help, describe commands) were falling back to %WINDIR% via GetTempPath() when TEMP/TMP/USERPROFILE were stripped from the environment. Affected probes could not execute without write access to the system directory.

Add Resolve-RunnerProbeTempRoot() to explicitly resolve a writable temp directory by consulting TEMP, TMP, and LOCALAPPDATA environment variables. Update New-RunnerProbeEnvironment to pin TEMP and TMP from the resolver, ensuring model-free probes always have OS scratch space without system directory fallback. Implement Get-RunnerSystemDirectorySet and Test-RunnerDirectoryWritable as supporting utilities.

Add regression test (test-probe-environment.ps1) covering both the probe temp resolution and the probe-vs-eval isolation boundary. Integrate into validate-skill-templates.ps1 as a required validation gate.

Align repository and runner documentation with the foreground Phase 1 handoff and its fail-closed execution contract.

Replace the durable supervisor path with a bounded foreground fan-out, tighten OpenCode session handling, and align package generation and validation with the revised execution boundary.

Keep lifecycle, conformance, orchestration, and finalization tests aligned with foreground dispatch and the updated isolation evidence.

Move Get-RunnerPreflightTimeoutSeconds from invoke-runner-owned-arms.ps1 to runner-common.ps1 as the authoritative source, making it the single point of truth for the fixed 120-second model-free preflight timeout. Update all callers in prepare-skill-evals.ps1 to use this centralized function. Fix PowerShell collection handling in runner-common to preserve empty HashSets on return. Add test coverage for collection preservation and variable preflight timeout calculations.

OpenCode discovery no longer filters models by availability. Instead, it mirrors every model exposed by all configured providers and returns availability only as presentation metadata. When OpenCode is selected without an explicit model, interactive preparation must present all discovered selectors and wait for the user's choice; the first, free, recommended, or previous model may not be selected automatically.

Remove the Get-PolicyName and Select-ModelsByPolicy functions that filtered OpenCode discovery to free models only. All harnesses now return all available models without filtering by availability. Update OpenCode CLI invocation from 'opencode models opencode --verbose' to 'opencode models --verbose' to avoid duplicating the provider name. This is a breaking change: OpenCode discovery now returns paid and unknown-availability models alongside free models.

Update test fixtures and assertions to validate OpenCode discovery now returns all models regardless of availability. Replace fixture model selectors with provider/model format (e.g., 'provider-free/free-model', 'provider-paid/Paid.Model'). Change OpenCode test expectations from filtering free-only to verifying all models are returned. Add test cases for paid and unknown-availability models. Add tests to verify the 'opencode models --verbose' command format. Add live-execution tests with a fake OpenCode CLI to validate provider and availability metadata are preserved. Remove the old test that expected failure when no free models were available.

Restructure the workflow to enforce a confirmation gate where the skill presents resolved parameters and explicitly asks the user to confirm before generating or writing the .snk file. Update evals to validate this behavior.

Fix OpenCode test assertions for platform-specific environment variables on non-Windows platforms. Fix PATH separator handling in model discovery. Add comprehensive validation rules for strong-name signing confirmation gate pattern.

Implement shared observability primitives for eval runners: a mandatory heartbeat cadence, lifecycle vocabulary, activity tracking that tees captured output to evidence files while exposing safe metadata, and a STDERR-only progress writer that never contaminates STDOUT protocol. Observability never blocks completion, never weakens watchdog enforcement, and never prints secrets or model content.

Pass observability context (runner id, phase, worker metadata) to Invoke-RunnerProcess for all model-process launches across Codex, GitHub Copilot, and OpenCode runners. Each runner's model-CLI lifecycle is now tracked and relayed to the shared observability system.

Add dedicated test module for observability regression testing and update existing runner tests to work with new progress tracking and activity streams. Coverage includes lifecycle state transitions, heartbeat emission, activity metadata accuracy, and STDERR relay correctness.

Add deterministic regression gate for runner observability and update Phase 1 validator to separate STDERR observability output from STDOUT machine protocol. This ensures heartbeats and progress signals never corrupt the summary JSON parsing, maintaining validator determinism.

Document the runner observability system: shared primitives, activity tracking, heartbeat semantics, lifecycle vocabulary, progress persistence, and security guarantees. Explain how observability maintains determinism while providing live feedback without blocking completion or weakening watchdog enforcement.

The freebuff runner is planned and blocked. Remove references from repository-level documentation to keep guidance current.

Add incremental stream activity tracking (events and bytes) to Invoke-RunnerProcess for live heartbeat metadata. Refactor stdout/stderr capture from ReadToEndAsync to CopyToAsync through activity streams. Add comprehensive observability tests for active inner processes, OpenCode streaming, Codex app-server protocol events, and synchronous preflight completion. Remove freebuff-readiness.md (planned/blocked runner). Update validator assertion for current runner support.

Refactor git-keep-a-changelog eval case 22 to reflect current runner support landscape. Replace Cline adapter references with OpenCode. Remove assertion for freebuff-readiness (planned runner). Update SKILL.md example to show GitHub Copilot and OpenCode as independently selectable runners instead of historical options.

Enhance runner-eval infrastructure with better heartbeat tracking and activity monitoring. Track lastProtocolActivityUtc in app-server protocol loops and emit it in progress events. Implement bounded heartbeat emission that prevents timeout stalls during I/O waits. Add outputDrainCompleted field to diagnose child process output draining. Remove stderr tail sanitization in favor of structured state tracking. Comprehensive test coverage for Codex app-server observability in both success and timeout scenarios.